[procmgr] Config gates for processes.d auto-start - #54732
[procmgr] Config gates for processes.d auto-start#54732jose-manuel-almaza wants to merge 26 commits into
Conversation
There was a problem hiding this comment.
Could this just read the config schema instead of duplicating everything ?
There was a problem hiding this comment.
I'd like that longer term, but afaik procmgr can't use the Go config runtime or the embedded schema. For config gates we only need env resolution for ~13 keys, so this PR keeps an explicit table that mirrors those schema env_vars (with schema comments cross-linking back). Happy to follow up with either schema-driven codegen for env_bindings.rs or a CI check that diffs ENV_BINDINGS against schema env_vars for gated keys. WDYT?
There was a problem hiding this comment.
The schema is a yaml schema with a well known format so I do think it could be used ?
Either read at build/run time, or to generate rust code for those configs
There was a problem hiding this comment.
Yep. That's what I meant by long term. I'll evaluate the effort to implement it now
There was a problem hiding this comment.
The reload path now leaves a failed config-managed process stopped even after a subsequent config reload supplies a valid command, removing the recovery behavior covered by the deleted test. The same diff also removes the schema defaults for sketch v3 routing and shadow settings while serializer code still reads them, which can produce an empty v3beta route or disable expected shadowing.
🤖 Datadog Autotest · Commit 8cc535a · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8cc535acbf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
🎯 Code Coverage (details) 🔗 Commit SHA: 10e542e | Docs | Datadog PR Page | Give us feedback! |
2ad9173 to
8fa88c5
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d2e1a30819
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Gate evaluation stays stubbed in the spawn-profiles PR; integration tests that require real YAML gate behavior belong in jose/procmgr-config-gates.
a06eeda to
df5ab47
Compare
Add an OperationGate so config reload and daemon shutdown block list, describe, get_status, get_config, create, start, and stop with FAILED_PRECONDITION while in progress.
Parse condition_config_any from processes.d and evaluate config gates before auto-start; stub always-open gates are replaced in the next commit.
…R 2/4) Restore core Agent SCM env cache, resolve_executable, tokio baseline env, secret-backend win32 helpers, and Windows to_command_stdio helpers moved out of the spawn-profiles PR so each stacked PR carries only what it uses.
Add Agent-parity config gate evaluation (YAML, env, fleet, system-probe derivations) so processes.d definitions can gate on process_config and related settings. Secret backend resolution lands in PR 3.
Revert accidental sketches-to-series rename in core_schema.yaml from the config-gates split. Serializer shadow settings still read sketches.* keys.
Mirror condition_path_exists behavior so ${DD_CONF_DIR} and other
placeholders resolve before YAML gate evaluation.
serde_yaml leaves yes/on as strings while go.yaml.in/yaml/v2 treats them as true. Apply YAML 1.1 parsing for file lookups only; env stays ParseBool.
Gate integration tests live in #54732 with the real evaluator after they were removed from the spawn-profiles foundation PR.
Restore pkg/config/setup/BUILD.bazel comments from the base branch. The #53249 split had added nonexistent schema.codegen flags; this PR only needs the config_windows_test.go test target wiring.
…g gates Use sp_bool_default for discovery.enabled so system_probe_config.enabled derivation matches Go on Linux (true when unset) and ECS Fargate (false).
Resolve fleet policy directory from the config file being evaluated (system-probe.yaml or datadog.yaml), matching Go system-probe loading.
…g gates Load YAML with saphyr so YAML 1.1 bool spellings apply only to plain scalars; quoted strings use ParseBool only, matching agent config semantics.
PATHEXT defaults use uppercase extensions while created files often use lowercase; match existing files via read_dir when Path::is_file fails.
…ails Warn and reuse the inherit/NUL stdio path when CreateFileW cannot open a configured stdout/stderr file, matching Unix and tokio spawn behavior.
Tag queued retries with the originating process UUID and resolve the live process by incarnation so a timer from a removed process cannot spawn a replacement with the same name.
Move had_successful_run onto ManagedProcess so set_config can reset RestartTracker without dropping post-success retry classification, and add tests for reload plus failed respawn retry chains.
Drop module and API docs that restate obvious behavior; keep short notes where Win32 spawn/auth semantics are non-obvious.
Treat config-changed reloads like a fresh PM start: reset counters on set_config, start running/failed auto-start processes under boot gates, and discard all pending crash timers from before the reload.
Align restart-policy log assertion with the current message format and resolve rustfmt/clippy failures on the spawn-profiles branch.
Restore process.rs, spawn/mod.rs, and shutdown.rs from the reload-gate branch so profile/user display and the unified stop path survive the config-gates rebase onto main.
…ofiles Keep config-gates on stop().await while adopting the ProcessManager reload helper split from spawn-profiles, and drop the obsolete align-reload commit.
Replace outdated module comments that referenced hand-written Go BindEnvAndSetDefault setup with schema YAML env_vars and schema.codegen.
Remove leftover merge markers in may_auto_start and align config-gate integration tests with RuntimeHandles after the manager module split.
What does this PR do?
Adds config gates to dd-procmgr so
processes.ddefinitions can usecondition_config_anyto auto-start only when Agent config says they should.Implementation mirrors the Windows legacy SCM startup checks in
dependent_services_windows.goand Agent config resolution:DD_*) with Agent precedence (ignore empty values, no trim beforeParseBool, legacyprocess_config.enabledtransforms)system_probe_config.enabled(USM/NPM/security knobs, sk-tracer and discovery adjustments)DD_*overrides from the core Agent SCMEnvironmentregistry when not set in the procmgr process envWires gate evaluation into
ManagedProcessstart/reload paths in the manager.Stack context: PR 2/4 split from #53249. Builds on #54731 (
jose/procmgr-spawn-profiles).ENC[...]secret backend resolution lands in PR 3 (jose/procmgr-secret-backend-gates). Process-agent dual-mode integration lands in PR 4 (jose/procmgr-windows-process-agent).Motivation
Moving subservices (starting with process-agent) to dd-procmgr requires the supervisor to apply the same start/stop rules as the Agent today. Without config gates, a
processes.dentry would always spawn when registered, which breaks parity with legacy SCM and fleet policy.Describe how you validated your changes
pkg/procmgr/rust/src/config_gate/(env bindings, YAML load, system-probe derivations, gate evaluation)pkg/config/setup/config_windows_test.go)Additional Notes
ENC[...]/ secret backend resolution in this PR (fleet policyENC[...]values are not resolved here either).pkg/config/schema/yaml/process_config.yamlfor env binding documentation.pkg/system-probe/config/so procmgr gate derivations stay aligned with Goadjust*logic.